Method and system for brain activity signal-based treatment and/or control of user devices

ABSTRACT

A method for characterizing a brain electrical signal comprising forming a temporo-spectral decomposition of the signal to form a plurality of time resolved frequency signal values, associating each instance of the signal value with a predetermined function approximating a neurological signal to form a table of coefficients collectively representative of the brain electrical signal.

REFERENCE TO CO-PENDING APPLICATIONS

The entire subject matter of U.S. Provisional application 62/046,078,filed Sep. 4, 2014, entitled METHOD AND SYSTEM FOR IMPROVED BRAINACTIVITY SIGNAL ANALYSIS, and PCT application PCT/CA2015/050839 filedSep. 2, 2015 entitled METHOD AND SYSTEM FOR BRAIN ACTIVITY SIGNAL-BASEDTREATMENT AND/OR CONTROL OF USER DEVICES is incorporated herein byreference including materials originally submitted to the U.S. PatentOffice.

FIELD OF THE INVENTION

The present invention relates to user treatments or controlling userdevices, based on analysis of brain activity signals.

DESCRIPTION OF THE RELATED ART

Current statistics indicate that there are more than 7 million people inthe United States who have survived a stroke or brain attack and areliving with the after-effects. A large number of these survivors areafflicted by severe upper limb paralysis, and some of these severelyparalyzed stroke patients will not respond to conventional therapy andwill require long-term assistance. Generally, stroke is caused byhemorrhage (in 15% of cases) or occlusion (in 85% of cases) of a bloodvessel in the brain; creating a lesion and localized neuronal death. Thebrain's ability to regenerate or repair a neural structure damaged bystroke is limited, therefore strokes which affect the sensorimotorcortex can cause permanent motor deficits in the side of the bodycontralateral to the affected cerebral hemisphere, a condition known ashemiplegia. Specifically, 70-85% of individuals are hemiplegic followingtheir first stroke, and 60% will be unable to independently performsimple activities of daily living (i.e., washing, dressing, andtoileting) six months after the event. In response to the bleakprognosis facing stroke patients, research has focused on developingdifferent methods of therapy which emphasize neurological recovery.

One such method is physiotherapy which aims to increase a patient'sfunctional ability using strengthening exercises, passive movements, andneuro-developmental approaches; and another method is occupationaltherapy which is focussed on improving skills relevant to a specifictask and/or developing compensatory strategies. However, motor recoveryobserved with physiotherapy and occupational therapy typically plateausin the first six months following stroke.

Functional electrical stimulation (FES) therapy is yet another methodwhich has been used successfully to restore both arm and hand functionin stroke patients with severe hemiplegia. This intervention requires atherapist to identify the patient's intent to move their paretic orparalyzed limb, and trigger electrical stimulation which facilitatesmovement of the same affected limb. The combination of the neuralactivity (i.e. motor planning) with the afferent input from theresulting movement (caused by electrical stimulation) appears tofacilitate positive neuroplastic changes resulting in restoration ofvoluntary movement. However, reliance on the therapist to determine thepatient's intention to move has several drawbacks with respect to FEStherapy. One such drawback is that there is reduced certainty that thepatient is actually attempting the movement which is stimulated with FEStherapy, or that the patient is actually attempting a movement at all;which makes involvement of the central nervous system uncertain. Anotherdrawback is that the time between the attempt and the delivery of thestimulation, a critical aspect of neuromotor rehabilitation associatedwith neuroplasticity, may fall outside the latency for optimal recovery,assuming the correct movement was attempted.

Attempts have also been made to develop effective brain computerinterfaces to sense and patient's intended action and deploy aprosthetic device to carry out the identified intended action by apatient. However, these attempts have seen limited outcomes.

It is an object of the present invention to mitigate or obviate at leastone of the above-mentioned disadvantages.

SUMMARY

In one aspect, there is provided a method for characterizing a brainelectrical signal comprising forming a temporo-spectral decomposition ofthe signal to form a plurality of time resolved frequency signal values,associating each instance of the signal value with a predeterminedfunction approximating a neurological signal to form a table ofcoefficients collectively representative of the brain electrical signal.

In another aspect, there is provided a method for controlling a devicebased on a recorded intent of a user, the method comprising:

-   -   a. characterizing a brain electrical signal signifying the        intent according to the method as defined herein; and    -   b. causing the device to perform the action.

In some exemplary embodiments, the device may be one of a robotic arm ordevice, full limb prosthesis, partial limb prosthesis, neuroprosthesisor functional electrical stimulation (FES) device that actuates aparalysed limb, and an orthotic device, among others.

In some exemplary embodiments, the causing step may include applying afunctional electrical stimulation (FES) treatment (or therapy) to theuser to trigger a specific user action, according to the recorded userintent.

In another aspect, there is provided a method of characterizing a brainelectrical activity signal emitted during a human activity, comprisingforming a plurality of frequency delineated signal segments, and foreach signal segment correlating an instance thereof with a functionapproximating a neuro signal associated with a neurocognitive orneuromuscular (or neurological) activity to form a series correlationvalues over time, and forming a time versus frequency array ofcorrelation values; and wherein the signal is an electroencephalographic(EEG) or an electrocorticographic (ECoG) signal.

Some exemplary embodiments may include associating a binary one or zeroto each of the correlation values according to predetermined criteria.

Some exemplary embodiments may include accumulating a number of arrays,each for an instance of a number of repeated human activities, andestablishing an incidence value for each element in the array. Theincidence value may be an average value or a probability measurerelative to a predetermined array value.

In another aspect, there is provided, in a system, acomputer-implemented method for creating numerical and visualrepresentations of brain activities by detecting and analysing transientactivity of at least one brain electrical signal, said method havinginstructions stored in a computer-readable medium and executable by aprocessing structure to cause said processing structure to at least:

-   -   a. form a temporo-spectral decomposition of the signal to form a        plurality of time resolved frequency signal values; and    -   b. associate each instance of the signal value to a        predetermined function approximating a neurological signal to        form a time frequency table of coefficients, the table        collectively representative of the signal.

In another aspect, there is provided, in a system, acomputer-implemented method for creating numerical and visualrepresentations of brain activities by detecting and analysing transientactivity of at least one brain electrical signal, said method comprisinginstructions stored in a computer-readable medium and executable by aprocessing structure to cause said processing structure to at least:

-   -   a. form a temporo-spectral decomposition of the signal to form a        plurality of time resolved frequency signal values;    -   b. associate each instance of the signal value to a        predetermined function approximating a neurological signal to        form a time frequency table of coefficients, the table        collectively representative of the signal; and    -   c. wherein said brain electrical signal is an        electroencephalographic (EEG) or an electrocorticographic (ECoG)        signal.

In some exemplary embodiments the brain electrical signal is a pre-motorsignal. Further, the the brain electrical signal may be detected andanalysed using a brain-computer interface (BCI). The brain-computerinterface may comprise an electrode array having electrodes forplacement on a subject at predetermined positions.

In some exemplary embodiments, the brain electrical signal comprisesdata signifying an intended neurocognitive or neuromuscular event. Thedata may be associated with stored data templates each representative ofneuro signals associated with neurocognitive or neuromuscular events toidentify the intended neurocognitive or neuromuscular event,respectively.

In some exemplary embodiments, the brain-computer interface isconfigured to issue a signal output with one or more instructions for anaction according to the identified intended neurocognitive orneuromuscular event for execution within a predetermined period.

In some exemplary embodiments, the action may be carried out by a realor virtual device. Examples may include a robotic arm, a full or partiallimb prosthesis, or an orthotic device, among others, or an electricallystimulated limb actuable by electrical stimulation.

In another aspect, there is provided a brain-computer interface (BCI),comprising a processing structure configured to at least:

-   -   a. receive a brain electrical signal from a subject, the signal        including data signifying an intended neurocognitive or        neuromuscular event;    -   b. form a temporo-spectral decomposition of the signal to form a        plurality of time resolved frequency signal values;    -   c. associate the data with stored data templates, each derived        from time resolved frequency signal values from template brain        electrical signals from the subject and representative of        template neurocognitive or neuromuscular events to identify the        intended neurocognitive or neuromuscular event; and    -   d. issue a signal output with one or more instructions for an        action, according to the identified intended neurocognitive or        neuromuscular event, to be executed within a predetermined        period of time from the input brain signal.

In another aspect, there is provided a brain-computer interface (BCI),comprising a processing structure configured to at least:

-   -   a) receive a first brain electrical signal from a subject, the        signal including data signifying a first intended neurocognitive        or neuromuscular event;    -   b) form a temporo-spectral decomposition of the signal to form a        plurality of time resolved frequency signal values for the first        intended neurocognitive or neuromuscular event;    -   c) associate the data with stored data templates each derived        from time resolved frequency signal values from template brain        electrical signals from the subject and representative of        template neurocognitive or neuromuscular events to identify the        first intended neurocognitive or neuromuscular event; and    -   d) issue a signal output with one or more instructions for an        action, according to the identified first intended        neurocognitive or neuromuscular event, to be executed within a        predetermined period of time from the brain electrical signal;    -   e) receive a second brain electrical signal from a subject, the        signal including data signifying a second intended        neurocognitive or neuromuscular event;    -   f) form a temporo-spectral decomposition of the signal to form a        plurality of time resolved frequency signal values for the        second intended neurocognitive or neuromuscular event;    -   g) associate the data with the stored data templates to identify        the second intended neurocognitive or neuromuscular event; and    -   h) issue a signal output with one or more instructions for an        action, according to the identified second intended        neuromuscular event, to be executed within a predetermined        period of time from the second brain electrical signal.

In some exemplary embodiments the action may include a neuroprosthesis,a functional electrical stimulation (FES) action, a robotic arm ordevice action, a prosthetic limb action, or an orthotic device action.

In some exemplary embodiments, the issuing steps are carried out in apre-motor phase and before a motor phase of a correspondingneurocognitive or neuromuscular event.

In another aspect, there is provided a data template for use with abrain-computer interface (BCI), the data template derived from timeresolved frequency signal values from temporo-spectral decompositions ofbrain electrical signal from a subject representative of aneurocognitive or neuromuscular event to classify an intendedneurocognitive or neuromuscular event for an action.

In some exemplary embodiments, the action including a real action or avirtual action. A real action may include a neuroprosthesis, afunctional electrical stimulation (FES) action, a robotic arm or deviceaction, a prosthetic limb action, or an orthotic device action.

In some exemplary embodiments the data template may generated bydetecting and analysing transient activity of at least one pre-motorbrain activity, such as an electroencephalographic (EEG) or anelectrocorticographic (ECoG) brain signal. The data template may bestored in a database having a plurality of other data templates, andwherein said database referenced to identify an unclassified brainsignal by comparing data associated with said unclassified brain signalto said data templates.

In another aspect, there is provided a method for characterizing a brainactivity signal corresponding to an intended activity (IA), comprisingrecording an event related desynchronization (ERD) signal of the IA,forming a temporo-spectral decomposition of the ERD signal to form aplurality of time resolved frequency ERD signal values, associating eachinstance of the ERD signal value to a function approximating a syntheticERD signal to form an ERD table of coefficients collectivelyrepresentative of the IA.

Some embodiments may include accumulating a number of ERD tables, eachfor an instance of an IA.

Some embodiments may include forming an ERD signature representationcharacterizing the IA, from the number of ERD tables for the IA.

Some embodiments may include associating an ERD table for anuncharacterized IA with the ERD signature representation of acharacterized IA to determine if the uncharacterized IA is an instanceof the characterized IA.

In some exemplary embodiments, the step of recording an ERD signal mayinclude collecting one or more electrode signals from one or moreelectrodes, with one or more ERD tables being associated with acorresponding electrode signal.

Some exemplary embodiments may further comprise forming the ERD tablefor a characterized IA from the number of ERD tables for the IA.

Some exemplary embodiments may further comprise forming a comparativeERD table for a plurality of characterized IA's.

Some exemplary embodiments may further comprise recording an ERD signalcorresponding to uncharacterized IA and forming an ERD table therefor.

Some exemplary embodiments may further comprise recording the ERD signalcorresponding to an uncharacterized IA for a number of successive timevalues and updating an ERD table for the uncharacterized IA for eachtime value. The time value may in some cases correspond to overlappingintervals and in other cases to non-overlapping time intervals.

Some exemplary embodiments may further comprise comparing the updatedERD table for the uncharacterized IA with the comparative ERD table todetermine if the uncharacterized IA is an instance of one of thecharacterized IA's.

Some exemplary embodiments may further comprise issuing an identitysignal after a minimum number of time values necessary to determine thatthe uncharacterized IA is an instance one of the characterized IA's.

Some exemplary embodiments may further comprise affirming that theuncharacterized IA is an instance one of the characterized IA's when apredetermined correlation count is achieved, each count corresponding toa correlation between corresponding segments of the ERD tables of thecharacterized IA's and the uncharacterized IA.

Some exemplary embodiments may further comprise advancing thecorrelation count when a minimum distance is recorded betweencorresponding segments of the ERD tables of the characterized IA's andthe uncharacterized IA.

Some exemplary embodiments may further comprise comparing an ERD tablefor an uncharacterized IA with the ERD table for the characterized IA todetermine if the uncharacterized IA is an instance of the characterizedactivity.

In some exemplary embodiments, the issuing of the identity signal occursbefore an expiry of a pre-motor phase of an action corresponding to theIA.

Some exemplary embodiments further comprise issuing an action signal inresponse to the identity signal, to initiate the action corresponding tothe IA, and before the expiry of the pre-motor phase of the actioncorresponding to the IA.

In some exemplary embodiments, the time duration between the actionsignal and the expiry of the pre-motor phase, is minimized and/oroptimized.

In some exemplary embodiments, the brain activity signal may originatefrom an electrical signal, a magnetic signal, or a chemical signal. Inthe case of an electrical signal, the brain activity signal may be anelectroencephalographic (EEG) or an electrocorticographic (ECoG) signal.

In another aspect, there is provided a method for controlling a userdevice function based on a recorded ERD signal of a user, the methodcomprising characterizing an uncharacterized intended activity (IA) asdefined herein, and issuing a signal to activate the user devicefunction according to the characterized IA.

In some exemplary embodiments, the user device is one of a robotic arm,full limb prosthesis, partial limb prosthesis, or a user treatmentdevice, a neuroprosthesis or functional electrical stimulation (FES).

In another aspect, there is provided, in a system, acomputer-implemented method having instructions stored in acomputer-readable medium and executable by a processing structure tocause said processing structure to carry out a method as defined herein.

In another aspect, there is provided a brain-computer interface (BCI),comprising a processing structure configured to at least:

-   -   a. form the ERD table as defined herein; and    -   b. issue a signal output for activating a user device function.

In another aspect, there is provided an ERD template for use with abrain-computer interface (BCI), the ERD template formed from one or moreERD tables as defined herein and representative of a plurality ofcharacterized IA's.

In another aspect, there is provided an ERD template, stored on anontransient computer readable medium, for use with a brain-computerinterface (BCI), the ERD template formed from one or more ERD tables asdefined herein, and representative of a plurality of characterized IA's.

In another aspect, there is provided an ERD template for use with abrain-computer interface (BCI), the ERD template formed from one or moreERD tables as defined herein, and representative of a neurocognitive orneuromuscular event to classify an intended neurocognitive orneuromuscular event.

Some exemplary embodiments of the above-noted methods and systems may beused to classify brain activity signals, such as electroencephalographicsignals, according to specific behaviours using a BCI. For example, aset of templates can be generated by repeating the above-noted methodsteps over several trials pertaining to the specific behaviour andaccumulating the results of all trials in a single histogram. A set oftemplates is generated for each one of the behaviours to be classified.It is also possible to compare the magnitude of the elements in thehistogram against a predetemined threshold and keep only those whichexceed the threshold either in their actual magnitudes or normalizedvalues. In order to classify a new electroencephalographic signal theabove-noted steps are applied, and for a distance based classifier, thedistance (Euclidean or any other suitable definition) is measuredbetween the correlation histogram (for the data to classify) and eachone of the correlation matrices for each one of the explored behaviours.

Advantageously, in some exemplary embodiments, the incorporation of aBCI into applications such as FES therapy in which a combined BCI andFES platform that derives a control signal from a non-affected,ipsilateral hemisphere may provide an alternate route of recovery forhemiplegic patients with abnormal or non-existent contralateralneurological activity. In some exemplary embodiments involving FES, aBCI able to classify EEG signals allows for the movement produced by FESto be consistent with the patient's motor intent. In addition, FES maybe triggered automatically by the BCI within a specified inter-stimulusinterval, improving the therapy's compliance with the conditionsrequired for paired associative stimulation dependent plasticity whichgoverns long-term potentiation (LTP) changes in the motor cortex. As anexample, a BCI and FES platform for stroke patients may be able tonavigate abnormal neurological activity that can result from a lesion.Therefore, a BCI controlled by ipsilateral (non-lesioned) motor signals,if paired with FES therapy, may provide a solution for patients withdeficits in their contralateral neural activity.

In another aspect, there is provided a system for enabling a userdevice, comprising at least one processor configured to run at least onecomputer program:

-   -   a) to record an event-related desynchronization (ERD) signal        received from an input in communication with a challenged user,        the ERD signal corresponding to an uncharacterized intended        activity (IA) of the user for a each time value of one or more        successive time values,    -   b) for each time value:        -   i. to access a comparative ERD table of coefficients for a            plurality of characterized IA's, and formed by normalizing            correlating a plurality of time resolved frequency ERD            signal values with a function approximating a synthetic ERD            signal;        -   ii. to compare the updated ERD table for the uncharacterized            IA with the comparative ERD table to determine if the            uncharacterized IA is an instance of one of the            characterized IA's; and    -   c) to issue an identity signal to activate the user device,        after a minimum number of time values necessary to determine        that the uncharacterized IA is an instance one of the        characterized IA's.

In some exemplary embodiments, the program is adapted to issue theidentity signal when a predetermined correlation count is achieved, eachcount corresponding to a correlation between corresponding segments ofthe ERD tables of the characterized IA's and the uncharacterized IA.

In some exemplary embodiments, the program is adapted to advance thecorrelation count when a minimum distance is recorded betweencorresponding segments of the ERD tables of the characterized IA's andthe uncharacterized IA.

In some exemplary embodiments, the program is adapted to issue theidentity signal before an expiry of a pre-motor phase of an actioncorresponding to the IA.

In some exemplary embodiments, the program is adapted to initiate theaction corresponding to the IA, and before the expiry of the pre-motorphase of the action corresponding to the IA.

In some exemplary embodiments, the brain activity signal originates froman electrical signal, a magnetic signal, or a chemical signal.

In some exemplary embodiments, the brain activity signal is anelectroencephalographic (EEG) or an electrocorticographic (ECoG) signal.

In some exemplary embodiments, the processor is adapted to receive thebrain activity signal from a plurality of operatively positionedelectrodes.

In some exemplary embodiments, the processor is adapted to receive thebrain activity signal from a single operatively positioned electrode.

In another aspect, there is provided a system for translating analogevent-related desynchronization (ERD) signals from a user into anidentifiable intended activity (IA), the system comprising:

-   -   a) at least one input to receive one or more ERD signals;    -   b) at least one output to send device action instructions to a        user device to carry out a device action corresponding to the        IA; and    -   c) a controller to communicate with the at least one input and        the at least one output of the user device, the controller        including at least one special purpose processor configured to        run at least one computer program:        -   i. to record an ERD signal received from the at least one            input, the ERD signal corresponding to an uncharacterized IA            of the user for each time value of one or more successive            time values,        -   ii. for each time value:            -   1. to access one or more ERD templates of coefficients                for one or more characterized IA's, the ERD templates of                coefficients being formed by correlating a plurality of                time resolved frequency ERD signal values with a                function approximating a synthetic ERD signal;            -   2. to update an ERD table for the uncharacterized IA and                to compare the updated ERD table with the ERD templates                to determine whether the uncharacterized IA is an                instance of one of the characterized IA's; and        -   iii. to initiate a corresponding device action instruction            on at the at least one output after a minimum number of time            values necessary to determine whether the uncharacterized IA            is an instance one of the characterized IA's.

In some exemplary embodiments, the the at least one computer program isconfigured to identify the IA in response to achievement of apredetermined correlation count, wherein each count corresponds to acorrelation between corresponding segments of the ERD templates of thecharacterized IA's and the updated ERD table of the uncharacterized IA.

In some exemplary embodiments, the at least one computer program isconfigured to advance the correlation count in response to a minimumdistance being recorded between corresponding segments of the ERDtemplates and updated ERD table of the characterized IA's and theuncharacterized IA respectively.

Some exemplary embodiments further comprise the user device, wherein theuser device includes a robotic arm or device, a full or partial limbprosthesis, an orthotic device, an electrical stimulation device, or aninterface to a virtual device.

In some exemplary embodiments, the at least one computer program isconfigured to initiate the device action instruction before the expiryof a pre-motor phase of a user action corresponding to the IA.

In some exemplary embodiments, the ERD signal is anelectroencephalographic (EEG) or an electrocorticographic (ECoG) signal,a magnetic signal, or a chemical signal.

In some exemplary embodiments, the at least one input includes a singleelectrode.

A method for translating analog event-related desynchronization (ERD)signals from a user into an identifiable intended activity (IA) to formone or more ERD templates for use in associating an updated ERD table ofan uncharacterized IA therewith to identify and initiate a correspondingintended device action by a user device, the method comprising:

-   -   a) recording one or more ERD signals from one or more electrodes        operatively placed on the user, wherein each of the ERD signals        corresponds to a characterized intended activity (IA) of the        user for each time value of one or more successive time values;    -   b) for each time value, forming a temporo-spectral decomposition        of the ERD signal, to form a plurality of time resolved        frequency ERD signal values;    -   c) associating each ERD signal value with a function        approximating a synthetic ERD signal to form at least one ERD        table of coefficients collectively representative of the        characterized IA;    -   d) repeating step c for a number of instances of the        characterized IA to form a number of at least one ERD tables;        and    -   e) forming the ERD template for the characterized IA from the        number of ERD tables for the characterized IA.

In some exemplary embodiments, the recording of the one or more ERDsignals includes collecting one or more electrode signals from one ormore electrodes, with one or more of the ERD tables being associatedwith a corresponding electrode signal.

In some exemplary embodiments, the time values correspond to overlappingor non-overlapping time intervals.

Some exemplary embodiments further comprise:

-   -   a) recording the ERD signal corresponding to an uncharacterized        IA for a number of successive time values and updating an ERD        table for the uncharacterized IA for each time value; and    -   b) associating the updated ERD table for the uncharacterized IA        with the ERD template of the characterized IA to determine        whether the uncharacterized IA is an instance of the        characterized IA.

Some exemplary embodiments further comprise issuing a device actioninstruction after a minimum number of time values necessary to determinewhether the uncharacterized IA is an instance of the characterized IA.

Some exemplary embodiments further comprise affirming that theuncharacterized IA is an instance one of the characterized IA's inresponse to achievement of a predetermined correlation count, whereineach count corresponding to a correlation between corresponding segmentsof the updated ERD table of the uncharacterized IA and the ERD templateof the characterized IA.

Some exemplary embodiments further comprise advancing the correlationcount in response to a minimum distance being recorded betweencorresponding segments of the updated ERD table and the ERD template.

In some exemplary embodiments, the device action instruction occursbefore an expiry of a pre-motor phase of an action corresponding to theaffirmed IA.

In some exemplary embodiments, the time duration between the deviceaction instruction and the expiry of the pre-motor phase, is minimizedand/or optimized.

In some exemplary embodiments, the user device is one of a robotic armor device, full limb prosthesis, partial limb prosthesis, or a usertreatment device, a neuroprosthesis or functional electrical stimulation(FES) device that actuates a paralyzed limb, an orthotic device, or avirtual activity device.

BRIEF DESCRIPTION OF THE DRAWINGS

Several exemplary embodiments of the present invention will now bedescribed, by way of example only, with reference to the appendeddrawings in which:

FIG. 1 is a top-level component architecture diagram of an exemplarysystem for processing brain activity signals;

FIG. 2 is top-level component architecture diagram of an exemplaryEEG-based BCI and FES system;

FIG. 3 shows a sensor glove indicating a change in output voltage at theonset of movement;

FIGS. 4a, 4b, 4c, 4d, 4e and 4f show six different hand movements;

FIG. 5 is an illustration of the sequence of visual cues;

FIG. 6a shows a raw EEG signal during a trial performance of a pinchgrasp movement;

FIG. 6b shows an optical sensor output from the trail performance ofFIG. 6 a;

FIG. 6c shows a sensor glove output during the trial performance of FIG.6 a.

FIG. 7 shows exemplary signals from the sensor glove and the opticalsensor recorded by the EEG amplifier before, during, and after executingthe pinch grasp and finger extension hand movements;

FIG. 8a shows superimposed hand movement data from the sensor gloveduring a plurality of exemplary trial pinch grasp movements;

FIG. 8b shows the same superimposed hand movement data from the sensorglove during a plurality of exemplary trial pinch grasp movements shownin FIG. 8a , after alignment with respect to the onset of movement;

FIG. 9 shows a high level flow diagram illustrating exemplary processsteps for classifying an exemplary brain activity signals;

FIGS. 10a and 10b are exemplary plots of the cross-correlation between a24 Hz EEG spectral component of a pinch grasp movement trial and asynthetic ERD signal;

FIG. 11 provides exemplary tables for nine trials of a pinch graspmovement showing time and frequency elements in which thecross-correlation exceeded a threshold;

FIG. 12 shows a histogram illustrating the average of all correlationdata for 30 trials of the pinch grasp movement of FIG. 12;

FIG. 13 shows exemplary histograms (‘all-in’ templates) of trials forsix exemplary movements;

FIG. 14 illustrates the process described by tables of FIGS. 15 and 16for the first five trials of the pinch grasp; and

FIG. 15 illustrates an exemplary method for characterizing an ERDsignal.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

The detailed description of exemplary embodiments of the inventionherein makes reference to the accompanying block diagrams and schematicdiagrams, which show the exemplary embodiment by way of illustration andits best mode. While these exemplary embodiments are described insufficient detail to enable those skilled in the art to practice theinvention, it should be understood that other embodiments may berealized and that logical and mechanical changes may be made withoutdeparting from the spirit and scope of the invention. Thus, the detaileddescription herein is presented for purposes of illustration only andnot of limitation. For example, the steps recited in any of the methodor process descriptions may be executed in any order and are not limitedto the order presented.

Moreover, it should be appreciated that the particular implementationsshown and described herein are illustrative of the invention and itsbest mode and are not intended to otherwise limit the scope of thepresent invention in any way. Indeed, for the sake of brevity, certainsub-components of the individual operating components, conventional datanetworking, application development and other functional aspects of thesystems may not be described in detail herein. Furthermore, theconnecting lines shown in the various figures contained herein areintended to represent exemplary functional relationships and/or physicalcouplings between the various elements. It should be noted that manyalternative or additional functional relationships or physicalconnections may be present in a practical system.

DEFINITIONS

To facilitate understanding of the disclosure, certain terms as usedherein are defined below. As used interchangeably herein, the terms“Functional Electrical Stimulation therapy” and “FES therapy” refer tothe application of electrical stimulation by a therapist,transcutaneously, to a paretic limb during the patient's consciouseffort to move the limb. Examples of FES systems are described in PCTapplication PCT/CA2011/000637 entitled FUNCTIONAL ELECTRICAL STIMULATIONDEVICE AND SYSTEM, AND USE THEREOF, the entire contents and subjectmatter of which are incorporated herein by reference.

As used herein, the term “brain activity” and “brain activity signal”refer to recordable signals generated by the brain, which may berecorded by way of electrodes or other sensors including those capableof sensing magnetic or chemical activity. Examples of brain activitysignals include brain electrical signals includingelectroencephalography (EEG), and electrocorticography (ECoG) recordedinvasively with subdural and/or epidural electrodes and the like.

As used interchangeably herein, the terms “brain-computer interface” and“BCI” refer to a platform which allows its operators to control aperipheral electronic device with activity of the brain.

As used herein, the term “event-related desynchronization” and “ERD”refers to a power decrease in a brain activity signal, such as an EEGsignal (among others), which occurs during motor planning and execution.In the case of an EEG signal event, the ERD typically occurs within thealpha (8-12 Hz) and beta (13-30 Hz) bands, though ERD characteristicsmay also occur at other frequencies or in other frequency ranges. An“ERD signal” refers to a signal which exhibits a measurable ERD.

As used herein, the term “synthetic ERD signal” refers to a waveformapproximating a naturally occurring ERD as defined by a mathematicalfunction.

FIG. 1 shows a top-level component architecture diagram of an exemplarysystem, generally identified by reference numeral 10, for detecting andclassifying distinct spectral (frequency), temporal (time) or otherfeatures of an ERD signal using an EEG-based BCI, though ERD signals mayalso be in other forms, such as electrocorticography (ECoG) recordedinvasively with subdural and/or epidural electrodes. With reference toFIGS. 1 to 3, an ERD signal, in this case in the form of an EEG signalis acquired using an electrode array 12, positioned on a participant'shead 13 to sense the brain's activity. A sensor glove 14, worn on theparticipant's hand, includes a resistive sensor for detecting the onsetof a neuromuscular event, in this case resulting in a hand movement. Adisplay monitor 16 presents visual cues to participants, and an opticalsensor 18 positioned inconspicuously on display monitor 16 recordsvisual cues to identify the stage of the experiment during dataanalysis. Signals from both the sensor glove 14 and the optical sensor18, along with the electrode array 12 are recorded using an amplifier20, such as the SynAmps RT EEG amplifier, available from Neuroscan,North Carolina, U.S.A. and provided as an input into acquisitioncomputer 22 employing at least one application program, such as CURRY 7acquisition software, available from Neuroscan, North Carolina, U.S.A.Signals from the sensor glove 14 and optical sensor 18 are also providedas inputs into acquisition computer 22. The signals may be provided toacquisition computer 22 either directly through lead wires or indirectlythrough a wirelessly transmitted signal. The ERD signals are interpretedand certain features are extracted therefrom using at least onealgorithm to generate ERD tables or data templates corresponding to theintended neuromuscular events, which relate to intended activities (IA),such as to include intended firing of muscles and/or muscle groups formovements of parts of the body, such as arms, legs, fingers and toes.The features are classified by correlating them against a functionrepresenting a synthetic ERD signal. A main characteristic of the ERD isthat it decays or decreases over time, which means that functionsoperably suited include those which decay or decrease over time, such asnonlinear tangent functions. Other functions, however, may also be used,including linear functions. In a combined BCI and FES system 30, datatemplates are stored in a template database 32 which is queried byacquisition computer 22 to issue a signal output with one or moreinstructions to FES system 34. (The FES system 34 may in otherexemplified embodiments be replaced by user devices to perform anaction, such as a robotic arm, full limb prosthesis, partial limbprosthesis, an orthotic device, among others.) In the case of an FEStreatment, the instructions may cause the participant to perform aspecific action, or effect treatment, based on the identified intendedneuromuscular event or IA.

An exemplary experimental protocol for a study to classify particularhand movements using pre-motor EEG activity using the apparatuspresented above will now be described. In the study, thetemporo-spectral representation of the ERD signal, in this case an EEGsignal, corresponding to specific movements of a hand were correlatedwith a function representing a synthetic ERD. A measurable ERD signalmay be used to differentiate between states of movement and rest. Thepower decrease in the ERD signal occurs during motor planning andexecution. This change in power occurs most prominently in the centralregion of the brain and is therefore thought to be related to theactivity of the sensorimotor cortex. Given that the hand has one of thelargest cortical representations in the sensorimotor map, it providesenhanced EEG signal resolution, which may be used to sense an ERDsignal, whose features may be correlated with a representative syntheticERD function.

Fifteen able-bodied individuals were recruited to participate in thestudy. Of the fifteen participants, fourteen were right handed and sixwere female. The average age of the participants was 32 years old. Theparticipants were uniquely identified, and for the purposes of thisdescription the participants will be referred to as participant 1,participant 2, participant 3, up to participant 15.

An electrode array 12 with eight electrodes was placed on theparticipant's head at the following EEG sites: C1, C2, C3, C4, CZ, F3,F4 and FZ (according to the international 10-20 system of electrodeplacement), as shown in FIG. 2. All electrode sites, reference (linkedear lobes) and ground (clavicle bone) were prepared with 70% isopropylalcohol and Nuprep® Skin Prep Gel, available from Weaver and Company,Aurora, Colo., U.S.A., prior to electrode placement with Ten20®Conductive EEG Paste, available from Weaver and Company, Aurora, Colo.,U.S.A. The impedance at each EEG site was measured, and preferably theimpedance had a value of less than 10 Detected signals from the eightEEG electrode array 12 were passed through a high-pass filter andsampled thereafter. In one example, a cut off frequency of 0.15 Hz andsampling frequency of 1 kHz was selected. This sampling frequency waschosen to promote temporal resolution and to the increase number of datapoints, since the signal analysis method may be applied off-line.

Participants were then asked to don a custom-made sensor glove 14 whichdetected the onset of hand movement using a resistive sensor. FIG. 3shows a sensor glove 14 indicating a change in output voltage at theonset of movement. The optical sensor 18 was placed against the lowerleft corner of the display monitor 16 and used to record visual cues,which were concealed from the participant and used to identify the stageof the experiment during data analysis. As described above, the signalsfrom the eight EEG electrodes were recorded using amplifier 20 andacquisition software running on acquisition computer 22, along withsignals from both the sensor glove 14 and optical sensor 18.

At the beginning of the session, participants were given instructions toperform six different hand movements including: non-functional 1movement (FIG. 4a ), palmar grasp (FIG. 4b ), non-functional 2 movements(FIG. 4c ), finger extension (FIG. 4d ), pinch grasp (FIG. 4e ), andlumbrical grasp (FIG. 4f ). These movements were chosen based on theirrelevance to post-stroke rehabilitation (i.e., finger extension, pinchgrasp, lumbrical and palmar grasps) as well as two non-functional graspswhich were intended to provide additional test cases for the study.

In each trial, the participants performed one of the specified six handmovements during a specified time interval. Visual cues presented ondisplay 16, including ‘ready’, ‘go’ and ‘stop’, were used to indicatethe stage of the trial to the participants, as shown in FIG. 5, andtheir meanings were explained to the participants prior to commencingthe experiment. (FIG. 5 represents examples of separate activities usingthe examples of FIGS. 4d and 4a respectively.) For example, at thebeginning of each trial, the participants were asked to relax for 10seconds while focusing on a white fixation cross as shown at FIG. 5a .The purpose of this interval was to allow participants to focus on theexperiment and disengage from external or environmental distractions.Following the relaxation interval, the participants were presented witha predetermined sequence of visual cues at predetermined time periods.For example, a yellow circle (represented by hatch markings in FIG. 5b), presented at time 1.0 to 3.5 seconds, indicating to the participantsthat a hand movement is about to be presented. Next, a picture of thehand movement to be performed (FIG. 5c ) was presented at time 3.5 to5.0 seconds, followed by a black screen (FIG. 5d ) presented at time 5.0to 7.0 seconds. Next, a green circle (represented by hatch markings inFIG. 5e ) was presented at time 7.0 to 7.5 seconds, indicating the handmovement to be performed, followed by a black screen (FIG. 5f )presented during the execution of the hand movement at time 7.5 to 9.5seconds. Finally, a red circle (represented by hatch markings in FIG. 5g) was presented at time 9.5 to 10.0 seconds, indicating to theparticipant to relax their hand. The sequence was repeated for eachprescribed hand movement.

In order to minimize participant fatigue the total experimental time wasseparated into three 6 minute experiments followed by three 5 minuteexperiments wherein the six hand movements were presented in a randomorder. The three longer experiments (6 minute) were completed firstsince participant fatigue generally increased with the duration of theexperiment. Each participant was given the opportunity to rest betweeneach experiment.

The participants completed the hand movements with their self-identifieddominant hand, except for four of the participants who repeated theexperiment with their non-dominant hand during a separate session.Generally, the EEG data collected during dominant hand movements wasexpected to contain more distinguishable features for classification,since the dominant hand has a larger sensorimotor representationrelative to the non-dominant hand. The data collected from theparticipants using their non-dominant hand was used to measure therobustness of the signal analysis approach developed for this study. Inboth scenarios, (the dominant or non-dominant hand experiment), each ofthe six movements were performed an average of 30 times; this samplesize allowed for successful movement classification to be reportedwithin a confidence interval of approximately +/−18% and a confidencelevel of 95%.

As noted above, signals from the electrode array 12 with the eight EEGelectrodes positioned at EEG sites: C1, C2, C3, C4, CZ, F3, F4 and FZwere recorded for each participant as they performed each of theprescribed hand movements. The optical sensor 18 recorded a sequence ofvisual cues which indicated both the stage of the experiment and thetype of hand movement depicted, while the sensor glove 14 detected thetype of hand movement. FIG. 6b shows the optical sensor 18 output. FIG.6c shows sensor glove 14 output during the performance of the pinch.Since each of the six hand movements were distinct, the output of thesensor glove 14 was characteristic to each movement, therefore anytrials in which the participant executed the incorrect grasp wereidentified visually and eliminated from the data. FIG. 6b shows anincrease in voltage recorded between −3.5 and −2 seconds correspondingto the time when the participant is viewing the hand movement to beperformed. The second voltage increase illustrated in this graph(occurring at approximately 0 seconds) corresponds to the time when theparticipant receives the instruction to execute the prescribed handmovement (green circle in FIG. 5).

FIG. 7 shows exemplary signals recorded by the EEG amplifier 20 duringan experiment, before executing the prescribed hand movement and whileexecuting said prescribed hand movement, namely the pinch grasp movement(in rows 1 to 3) and a finger extension hand movement (rows 4 to 6).FIG. 8a shows a graph of the output of the sensor glove 14, where theplots of 30 trials are superimposed (with the dashed line representingaverage values). FIG. 8b shows the same data after alignment, asdiscussed below. As can be seen from FIG. 8b , the graph, the onset ofhand movement is indicated by the decrease in voltage initiated at 0seconds. Accordingly, as shown in FIG. 7, it is evident from both theoptical sensor 18 output and the sensor glove 14 output that the handmovement was performed shortly after viewing the green circle, asdescribed with reference to FIG. 5, instructing the participant toexecute the prescribed hand movement.

The collected EEG data was inputted into a Matlab® application program,available from MathWorks, Natick, Mass., U.S.A., running on acquisitioncomputer 22. The application program included coded instructions toeliminate trials in which the incorrect movement or no movement wasperformed. As was previously described, the type of movement expectedand performed was determined using both the sensor glove 14 and opticalsensor 18. Trials in which correct movements were performed were groupedaccording to the movement and aligned to the onset of movement. FIG. 8ashows hand movement data from the sensor glove 14 worn by theparticipants during experiment prior to alignment, FIG. 8b shows thesame data after alignment. The term “alignment” refers to a process ofidentifying a specific landmark, in this example a change of voltage inthe sensor of sensor glove 14 indicating the onset of movement, and thenshifting the signals so that this event is, in each curve, aligned witha common instance of time. This enables the data prior to the landmarkto be pre-motor activity for curve analysis purposes, and the data afterthe landmark to the data after the onset of movement. In this case, thisenables the same amount of data to be available for the pre-motoractivity, that is prior to the onset of movement. In the example of FIG.8b , the plots are “aligned” to the landmark defined by the 4^(th)second instance of time of the resulting “aligned” signal, whichcorresponds to the 4000^(th) sample. Seven seconds of each trial wereextracted for further analysis which included the 4 seconds prior tomovement and the 3 seconds following the onset of movement.

The EEG signal was characterized by following the exemplary method stepsshown in a flow chart of FIG. 9.

In step 100, during preliminary analysis of each dataset,temporo-spectral decomposition of each trial was performed using a fastFourier transform with an exemplary Hamming window of length 256,overlap of 50% and a resolution of 1 Hz for frequencies between 1 and 50Hz, resulting in a spectrogram (time-frequency) representation of thesignal to be analyzed, such as a 72×50 (time-frequency) matrix(spectrogram).

Next, in step 102, each of the time-resolved frequency components (from1 Hz to 50 Hz) in the dataset was normalized and smoothed using a movingaverage filter (for example, with a window size of 10).

A synthetic ERD function similar to the general morphology of thenaturally occurring ERD event was subsequently determined to provide asynthetic ERD signal, and represented using a hyperbolic tangentfunction:

ERD_(syn)=−(tan h(4×)/3)  (Equation 4.1)

Equation 4.1 approximates the general morphology of the naturallyoccurring ERD event, which is characterized by a power decrease of theEEG within discrete frequency bands of alpha (10-12 Hz) and beta (13-25Hz) preceding and during voluntary movement.

Next, in step 104, a cross-correlation between each one of thetime-resolved spectral components and the synthetic ERD function wascalculated, to generate a matrix (spectral components versus timeinstances) with correlation values. For example, cross-correlationcoefficients between each of the time-resolved frequency signals, from 1to 50 Hz and the synthetic representation of an ERD were determined withthe following exemplary equation:

$\begin{matrix}{{{{R\left( {i,\left( {{ERD}_{syn},f_{j}} \right)} \right)} = \frac{C\left( {{ERD}_{{syn},}f_{j}} \right.}{\sqrt{{C\left( {{ERD}_{syn},{ERD}_{syn}} \right)} \cdot {C\left( {f_{j},f_{j}} \right)}}}};}\mspace{79mu} {j \in {\left\lbrack {1,2,\ldots \mspace{14mu},50} \right\rbrack i} \in \left\lbrack {1,2,\ldots \mspace{14mu},20} \right\rbrack}} & \left( {{Equation}\mspace{14mu} 4.2} \right)\end{matrix}$

where R refers to a matrix of cross-correlation coefficients between thesynthetic ERD signal (ERD_(syn)) and a time-resolved frequency signal(f_(j)), where jε[1, 2, . . . , 50] for each time instance, iε[1, 2, . .. , 20]. C(ERD_(syn), f_(j)) is the covariance between the two signals,ERD_(syn) and f_(j); C(ERD_(syn),ERD_(syn)) is the variance of the ERDsignal, and C(f_(j),f_(j)) is the variance of a time-resolved spectralcomponent. Equation 4.2 was applied to segments of each time-resolvedspectral component, which were 20 data points in length, at 20 instancesprior to the onset of movement with an overlap between segments of 19data points. Each of the times instances, from 1 to 20, corresponds to atime prior to the onset of movement as illustrated in Table 1. Forgreater clarity, FIGS. 10a and 10b illustrate all of the instances inwhich the cross-correlation is applied. In this case, the majority ofthe downward slope of the EEG signal (which actually contains the mostinformation and therefore is most relevant for the detection process) isbefore the onset of the movement. FIG. 10b shows the application ofEquation 4.2 for all 20 time intervals, for a 23 Hz smoothed andnormalized time-resolved spectral component (represented in black ink)recorded during a pinch grasp, in which movement onset occurs at 0seconds. Represented in solid lines are the multiple instances of thesynthetic ERD signal where cross-correlation coefficient werecalculated.

In step 106, thresholding was applied to the result of each sequence ofcross-correlation calculations according to the following criterion:

G(i,j,k)_(n)=1 for R _(i,j) ≧n and 0 f or R _(i,j) <n; iε[1 20], jε[150], k=number of trials  (Equation 4.3)

where G_(n) contains binary values of correlations which exceed aspecific threshold: n=[0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9].

FIG. 11 shows the distribution of significant cross-correlations (r>0.8and p<0.05) for the first nine trials of the pinch grasp and syntheticERD signals, when the thresholding process is applied to the EEGactivity recorded at C3 from one participant. The areas in whiterepresent the times prior to the onset of movement when the correlationbetween each time-resolved frequency signal and the synthetic ERDexceeded the threshold value, such as 0.8, while the areas in blackrepresent a correlation value less than the threshold value. Thisprocess was completed for each trial using every threshold valuespecified. Predictably, lower threshold values resulted in increasedareas of white, while higher threshold values contain less areas ofwhite.

For each grasp, an average was calculated for each location in thematrix (step 108). FIG. 12 shows the “all-in” template with the averagesof all correlation data above a threshold of 0.8 for 30 trials of thepinch grasp recorded from the F3 position (‘all-in’ template). Warmercolors (W) represent a higher incidence of significantly correlatedarea, and cooler colors (C) represent a lower incidence of significantcorrelations. (As used herein, ‘all-in’ templates are so named as theyinclude every trial in the average.)

For a single electrode site, each participant had six ‘all-in’ templates(one for each hand movement), as shown in FIG. 13; resulting in a totalof 42 ‘all-in’ templates. In FIG. 13, warmer colors (W) represent ahigher incidence of significantly correlated area, and cooler colors (C)represent a lower incidence of significant correlations. Next, ‘one-out’templates were created by iteratively eliminating one trial andcalculating the average of the remaining trials; for a grasp executed 30times, 29 ‘one-out’ templates; represented by 20×50×29 (timeinstances-frequency-trial number) tensors were generated for eachthreshold value.

The afore-mentioned process may be used to classify brain activitysignals according to specific behaviours. This can be achieved bygenerating a set of templates by repeating steps 100 through 108 overseveral trials under the same behaviour and accumulating the results ofall trials in a single histogram. This process is repeated for each oneof the behaviours to be classified, and the templates are stored intemplate database 24. It is also possible to compare the magnitude ofthe elements in the histogram against a threshold and keep those (or adesignated sample thereof) which exceed the threshold either in theiractual magnitudes or normalized (set to one).

A new brain activity signal may be classified by applying steps 100through 108 and, for a distance based classifier, the distance(Euclidean or any other suitable definition) between the correlationhistogram (step 108) for the data to classify and each one of thecorrelation matrices for each one of the explored behaviours may bemeasured. An exemplified approach is described in more detail below.

The Euclidean distance between an ‘all-in’ template for a particularmovement (FIG. 12) and an individual trial (FIG. 11) of a different(uncharacterized) movement may be used as a measure of the similaritybetween the two movements. For example, using data from a singleelectrode site and correlation threshold, the distance between the firsttrial of the pinch grasp (grasp 1) and the average of all trials of thenon-functional 1 movement (grasp 2), may be calculated with thefollowing equation:

$\begin{matrix}{\mspace{79mu} {{{{D\left( {{\Lambda \; i},{\Lambda \; j}} \right)}_{1,2} = {{{G\left( {i,j,1} \right)}_{1} - \frac{\left( {\sum\limits_{k = 1}^{N_{{NF}\; 1}}{G\left( {i,j,k} \right)}_{2}} \right.}{N_{NF1}}}}};}\mspace{79mu} {{i \in \left\lbrack {1,\ldots \mspace{14mu},20} \right\rbrack},{j \in \left\lbrack {1,\ldots \mspace{14mu},50} \right\rbrack},{and}}{N_{{NF}\; 1} = {{the}\mspace{14mu} {number}\mspace{14mu} {of}\mspace{20mu} {trials}\mspace{14mu} {the}\mspace{14mu} {non}\mspace{14mu} {functional}\mspace{14mu} 1\mspace{14mu} {movement}\mspace{14mu} {was}\mspace{14mu} {executed}}}}} & \left( {{Equation}\mspace{14mu} 4.4} \right)\end{matrix}$

where D(Λi, Λj)_(1,2) is a matrix containing numerical values ofdistance between each element of the first trial of the pinch grasp,G(i,j,1)₁, and the average of all trials of the non-functional 1movement. Equation 4.4 is then applied to the first trial of the pinchgrasp and every ‘all-in’ average of the remaining four movements whichinclude: the lumbrical grasp, finger extension, the non-functional 2movement, and palmar grasp. When comparing an individual trial with thetemplate of the same movement, a ‘one-out’ template is used such thatthe individual trial being classified is not included in the averageused to create the ‘one-out’ template.

For example, in the earlier described experimental protocol, thefollowing equation was used when calculating the distance between Trial1 of the pinch grasp and the average of trials of this movement:

$\begin{matrix}{\mspace{79mu} {{{{D\left( {{\Lambda \; i},{\Lambda \; j}} \right)}_{1,1} = {{{G\left( {i,j,1} \right)}_{1} - \frac{\left( {\sum\limits_{k = 2}^{N_{Pinch}}{G\left( {i,j,k} \right)}_{1}} \right)}{N_{Pinch}}}}};}\mspace{79mu} {{i \in \left\lbrack {1,\ldots \mspace{14mu},20} \right\rbrack},{j \in \left\lbrack {1,\ldots \mspace{14mu},50} \right\rbrack},{and}}{N_{Pinch} = {{the}\mspace{14mu} {number}\mspace{14mu} {of}\mspace{20mu} {trials}\mspace{14mu} {the}\mspace{14mu} {pinch}\mspace{14mu} {grasp}\mspace{14mu} {was}\mspace{14mu} {executed}}}}} & \left( {{Equation}\mspace{14mu} 4.5} \right)\end{matrix}$

where D(Λi, Λj)_(1,1) is a matrix containing numerical values ofdistance between each element of the first trial of the pinch grasp(G(i,j,1)), and average of all trials of that movement (G(i,j,k)₁) withTrial 1 removed. The results of equations 4.4 and 4.5 were assembled ina 20×50×6 tensor containing numerical distances between Trial 1 of thepinch grasp and all other movements. Next, this tensor was summed alongthe 2^(nd) dimension (which refers to the frequencies included in theanalysis: 1-50 Hz), resulting in a 20×6 matrix. The minimum non-zerovalue at each time instance was then identified and assigned a value of1 and all other entries given a value of 0. For example, Table 2illustrates the actual values of distance calculated between Trial 1 ofthe pinch grasp and the ‘all-in’ template of each additional grasp(columns 3-7 of Table 2) and between Trial 1 of the pinch grasp and the‘one-out’ template of the pinch grasp (column 2 of Table 2) at each timeinterval prior to movement. Table 3 represents the binary version of thedata, where values exceeding the minimum distance in each row isassigned a value of 1, and all other distances are given a value of 0.

Zero values in the table shown in Table 3 are excluded from thecalculation of the minimum entry as these instances indicate thesubtraction of two zero values, meaning that neither instance resultedin a value of correlation with the synthetic ERD above the set threshold(Equation 4.3). Entries of 1 in the column labeled ‘Pinch’ (highlighted)indicate time intervals when Trial 1 of pinch had a minimum distancefrom the average for the pinch grasp relative to the average of theremaining grasps.

This process was then applied to every trial of the pinch grasp,resulting in N_(Pinch)×20×6 matrices. The percentage of all pinch grasptrials which were identified as having the minimum distance from thepinch grasp were then calculated. FIG. 14 illustrates the processdescribed by Tables 2 and 3 for the first five trials of the pinchgrasp, in which the areas in white represent the movement which had theminimum distance between Trial 1 of the pinch grasp at each timeinterval. The percentage of all pinch trials in which each of the sixmovements were identified to have the minimum distance is shown in FIG.14. The areas of lighter color refer to a higher percentage of pinchtrials with a minimum distance from a movement, and areas of darkercolor represent a lower percentage. Finally, the movement whichcontained the maximum percentage across all time intervals was selectedto classify the movement. In the example illustrated by FIG. 14, themaximum percentage is indicated by an ellipse; occurring in the firstcolumn, which is designated to the pinch grasp, during the time intervalof 1.56 to 0.88 seconds prior to the onset of movement with a maximumvalue of 71%. In other words, for this participant, the pinch grasp isclassified correctly in 71% of trials using this particular electrodeand correlation threshold during this specified time interval. Tocomplete the classification for the pinch grasp, the same proceduredescribed in this section was repeated for every electrode site (C1, C2,C3, C4, CZ, F3, F4, and FZ) and every value of correlation threshold(0.60, 0.65, 0.70, 0.75, 0.80, 0.85, and 0.90) resulting in 56 matrices(8 electrode sites×7 threshold values). Ultimately, the highestpercentage of classification achieved across any of the 56 matrices isselected to classify the movement. The remaining movements wereclassified using the same procedure.

FIG. 15 illustrates an exemplary embodiment of a method 120 forcharacterizing a brain activity signal corresponding to an intendedactivity (IA). In this case the method steps are carried out in advanceof T=To, that is the time signifying the initiation of a motor event,following the IA. First, at 122, an ERD table is accessed for a numberof characterized intended activities (CIA's), following from one or moreof the exemplified methods or protocols mentioned herein.

Next, at 124, at a given T=T1, an ERD signal is recorded for anuncharacterized intended activity (UCIA). At step 126, an ERD table isupdated for the UCIA for T1. At 128, a correlation count for each CIA isadvanced when a minimum distance is recorded between correspondingsegments of the ERD tables of the CIA's and the UCIA. Next, at 130, allthe correlation counts are compared against a predetermined minimumcount threshold, and if no count exceeds the threshold, then at 132, theERD signal is received for the next time increment. If any one countexceeds the threshold, then at 134 the UCIA is determined to be the CIAcorresponding to the threshold-exceeding count, and at 136, anactivation signal is issued, before T=To.

The steps 132, 136 and 136 may be carried out in real time, that is inthe time period of the pre-motor activity, that is between the instantthe IA signal is received and the instant at which the actioncorresponding to the IA is to be carried out. This means that the actualprocessing needed between receipt of the IA signal and the activationsignal may vary from one received ERD signal to the next, depending onthe nature of the IA. For instance, an ERD signal for moving a finger ina 90 degree path, in system that is capable of detecting the differencebetween a 90 degree movement and a 45 degree movement, may require moretime intervals to achieve the minimum correlation as the system isevaluating very slight differences in the ERD signals for both. Incontrast, if the system is only capable of recording a finger per se andnot sufficiently granular to distinction different finger movements mayachieve a minimum correlation count in a relatively shorter time period,when it is distinguishing between, for instance, finger movements versuswrite movements. Still further, the steps may be carried out in batchformat, that is they may be carried for a given number of timeintervals, which may be set to remain constant from one analysis to thenext.

Since the EEG data used for movement classification was limited to onlypre-motor activity, exemplified embodiments may be used to bothdifferentiate and predict the hand movement to be performed withreasonable accuracy.

In exemplary embodiments, analysis required for classification of eachtrial may be applied only to the EEG data recorded prior to the intendedactivity (IA), such as a hand movement by the participant. Thepre-movement interval may range from 2.5 seconds to 0 seconds prior tomovement and may be segmented into a number of discrete time steps, suchas 20 in the above example. The percentage of trials classified for eachof the six movements may be evaluated at each time step, and the highestpercentage may then be selected to classify the movement. In some cases,an intended activity may observed as early 1.5 seconds prior tomovement, though processing speeds in an online, synchronous, BCI andFES application may, in cases with suitable processing speeds, such asin the vicinity of 0.3 s or less, an exemplified method may beconfigured to detect and classify an ERD signal in time to trigger anappropriate response via FES. As a result, exemplified methods andsystems herein may be deployed to stimulate a volitional hand movementin the operator for the purposes of motor training; as discussedearlier, this approach may be successful in restoring motor control ofthe hand in stroke patients with hemiplegia. In other words, exemplifiedmethods and systems herein may be configured to characterize an intendedactivity and trigger an action to an FES treatment step or anotheraction in a user device in a real or virtual environment at or near anoptimal firing time, as can be configured according to conditionsappropriate for the application. Thus, exemplified methods and systemsmay be configured so that a time duration between an action signal andthe expiry of the pre-motor phase of the associated action, is minimizedand/or optimized, according to such factors as operational delays, asmay occur in prosthetic, orthotic, exoskeletal, robotic or otherautomated devices and the like, which may be configured to carry out arepresentation of, or for that matter operationally mimic, an intendedaction. For instance, some devices may require a period of latency forpreparation to a ready state in advance of action. Further, some usersmay encounter operational delays arising from some brain functionlimiting conditions.

For instance, a BCI may be implemented as a “brain-switch” to produce auser device instruction by way of one or more control signal, which maybe conveyed to the user device to execute a prescribed action, alongwith additional information in relation to the prescribed action, suchas coordinates for the placement of a prosthetic appendage in a targetconfiguration.

In the above exemplary protocol, the average time when each trial wassuccessfully classified ranged from 0.3 seconds to 2 seconds prior tomovement for the dominant hand; and 0.3 seconds to 1.4 seconds fornon-dominant hand movements across participants. The ERD signal in somecases was observed and the intended activity classified as early 1.5seconds prior to movement, and in one example was detected in real-timean average of 0.62 seconds before movement. In the above exemplaryprotocol, a maximum of eight EEG electrodes was used, which may besubstantially less than other prior methods which may requiresubstantially more electrodes and are not adaptable to classifydifferent hand movements using pre-motor activity. As such, the use ofthe eight EEG electrodes, makes it more viable for use in a clinicalsetting. That said, in some exemplary embodiments, operable results maybe achieved with data from a single electrode.

In exemplary embodiments, a set of parameters may be selected which maybe unique for each participant, or for a group of participants,depending on such variables including the type of hand movement, and thespatial location of electrodes. In yet another exemplary embodiment,methods and systems described above may be employed to createnon-invasive brain-computer interfaces with high communicationthroughputs (each identifiable behaviour represents a different commandavailable to the user).

In yet another exemplary embodiment, methods and systems described abovemay be employed to create brain-computer interfaces with a high level ofinteraction transparency if used to control a device to facilitatemovement of a paralyzed or nonexistent limb (e.g., a functionalelectrical stimulator).

In yet another exemplary embodiment, methods and systems described abovemay be employed to enhance therapies which facilitate a movement of aparalyzed limb using artificial/external means, such as functionalelectrical stimulation therapy, after patients attempt the movement forseveral seconds. For example, the afore-mentioned methods and systemsmay improve these therapies by 1) triggering the mechanism to producethe movement by identifying the intention to move through analysis ofbrain signals alone, 2) facilitating the specific intended movement, 3)providing a mechanism to ensure that patients are in fact attempting tomove, and 4) triggering the mechanism to produce movement withinphysiologically realistic latencies.

In yet another exemplary embodiment, the afore-mentioned methods andsystems may be employed to image brain activities, for example, byconducting analysis of neurological events of short duration which maylead to the discovery and characterization of new features correlatedwith behaviour and other neurophysiological events. Accordingly, theafore-mentioned methods and system may be integrated into new orexisting commercial software for the analysis of brain activities.

In yet another exemplary embodiment, the afore-mentioned methods andsystems may be employed as screening and/or diagnostic tools forneurological conditions based on the ability to identify transientevents in electroencephalographic (and potentiallyelectrocorticographic) signals. Accordingly, the afore-mentioned methodsand systems may be integrated into new or existing commercial softwarefor the analysis of brain activities.

In yet another exemplary embodiment, the afore-mentioned methods andsystems may be employed to create access methods for patients that areunable to use current assistive devices reliably. The resultingassistive technologies may have a high degree of transparency if theintended and executed actions correspond exactly or at leastoperatively, and/or may offer a number of options greater than what itis currently possible.

In yet another exemplary embodiment, the brain activity signal may be anelectrocorticographic (ECoG) signal.

Although the above-noted methods and systems have been described interms of humans, these methods and systems are applicable to animals.

Thus, exemplary embodiments provide technical utility by providing atechnical solution to the conventional technical problem of identifyingan IA, and in some cases a series of IA's in succession, in aquantifiable way, from one or more raw analog signals obtained from anelectrode array, in a reasonably timely and accurate manner, to enableeffective control of several external (e.g., virtual or real) actions ina synchronous manner to the actions to be taken as a result of theidentified IA or IA's. Furthermore, in some exemplary embodiments, theprovided technical solution may be to the problem of identifying the IA,and in some cases a series of IA's in succession from a singleelectrode, rather than an array of electrodes.

Thus, some exemplary embodiments utilize a special purpose computer forthis purpose, acting in a quantifiable and repeatable manner totranslate raw analog signals into quantifiable, identifiable and/ormappable IA's so as to enable control of an action device according tothe quantifiable, identifiable and/or mappable IA's. Accordingly, thepresently disclosed embodiments provide technical utility by enablingissuance of quantifiable, identifiable and mappable instructions to aprosthetic, neuroprosthetic, FES, robot, orthotic device, or to avirtual device.

The preceding detailed description of exemplary embodiments of theinvention makes reference to the accompanying drawings, which show theexemplary embodiment by way of illustration. While these exemplaryembodiments are described in sufficient detail to enable those skilledin the art to practice the invention, it should be understood that otherembodiments may be realized and that logical and mechanical changes maybe made without departing from the spirit and scope of the invention.For example, the steps recited in any of the method or process claimsmay be executed in any order and are not limited to the order presented.Further, the present invention may be practiced using one or moreservers, as necessary. Thus, the preceding detailed description ispresented for purposes of illustration only and not of limitation, andthe scope of the invention is defined by the preceding description, andwith respect to the attached claims.

TABLE 1 Interval Number Interval Start Interval End Mid-Point 1 −3.87−1.15 −2.51 2 −3.74 −1.02 −2.38 3 −3.60 −0.88 −2.24 4 −3.46 −0.74 −2.105 −3.33 −0.61 −1.97 6 −3.19 −0.47 −1.83 7 −3.06 −0.34 −1.70 8 −2.92−0.20 −1.56 9 −2.78 −0.06 −1.42 10 −2.65 0.07 −1.29 11 −2.51 0.21 −1.1512 −2.38 0.34 −1.02 13 −2.24 0.48 −0.88 14 −2.10 0.62 −0.74 15 −1.970.75 −0.61 16 −1.83 0.89 −0.47 17 −1.70 1.02 −0.34 18 −1.56 1.16 −0.2019 −1.42 1.30 −0.06 20 −1.29 1.43 0.07

TABLE 2 Time Non- Non- Prior to Func- Func- Movement Pinch tional 1Lumbrical Extension tional 2 Palmar −2.51 0.03 0.19 0.06 0.06 0.10 0.55−2.38 0.06 0.13 0.13 0.48 0.10 0.10 −2.24 0.48 0.16 0.03 0.06 0.10 0.16−2.10 0.03 0.16 0.06 0.61 0.06 0.06 −1.97 0.10 0.06 0.10 0.65 0.06 0.03−1.83 0.06 0.00 0.65 0.13 0.07 0.10 −1.70 0.03 0.13 0.00 0.71 0.06 0.06−1.56 0.13 0.13 0.03 0.10 0.00 0.61 −1.42 0.00 0.06 0.13 0.68 0.10 0.03−1.29 0.10 0.10 0.03 0.10 0.06 0.61 −1.15 0.68 0.03 0.00 0.06 0.06 0.16−1.02 0.71 0.06 0.06 0.06 0.06 0.03 −0.88 0.77 0.00 0.10 0.03 0.06 0.03−0.74 0.06 0.03 0.16 0.71 0.03 0.00 −0.61 0.13 0.77 0.03 0.06 0.00 0.00−0.47 0.03 0.84 0.06 0.06 0.00 0.00 −0.34 0.00 0.03 0.04 0.84 0.06 0.04−0.20 0.06 0.74 0.03 0.10 0.07 0.00 −0.06 0.68 0.13 0.10 0.03 0.04 0.040.07 0.55 0.06 0.19 0.07 0.13 0.00

TABLE 3 Time Non- Non- Prior to Func- Func- Movement Pinch tional 1Lumbrical Extension tional 2 Palmar −2.51 1 0 0 0 0 0 −2.38 1 0 0 0 0 0−2.24 0 0 1 0 0 0 −2.10 1 0 0 0 0 0 −1.97 0 0 0 0 0 1 −1.83 1 0 0 0 0 0−1.70 1 0 0 0 0 0 −1.56 0 0 1 0 0 0 −1.42 0 0 0 0 0 1 −1.29 0 0 1 0 0 0−1.15 0 1 0 0 0 0 −1.02 0 0 0 0 0 1 −0.88 0 0 0 0 0 1 −0.74 0 1 0 0 0 0−0.61 0 0 1 0 0 0 −0.47 1 0 0 0 0 0 −0.34 0 1 0 0 0 0 −0.20 1 0 0 0 0 0−0.06 0 0 0 1 0 0 0.07 0 1 0 0 0 0

1. A system for translating analog event-related desynchronization (ERD)signals from a user into an identifiable intended activity (IA), thesystem comprising: a. at least one input to receive one or more ERDsignals; b. at least one output to send device action instructions to auser device to carry out a device action corresponding to the IA; and c.a controller to communicate with the at least one input and the at leastone output of the user device, the controller including at least onespecial purpose processor configured to run at least one computerprogram: i. to record an ERD signal received from the at least oneinput, the ERD signal corresponding to an uncharacterized IA of the userfor each time value of one or more successive time values, ii. for eachtime value:
 1. to access one or more ERD templates of coefficients forone or more characterized IA's, the ERD templates of coefficients beingformed by correlating a plurality of time resolved frequency ERD signalvalues with a function approximating a synthetic ERD signal;
 2. toupdate an ERD table for the uncharacterized IA and to compare theupdated ERD table with the ERD templates to determine whether theuncharacterized IA is an instance of one of the characterized IA's; andiii. to initiate a corresponding device action instruction on at the atleast one output after a minimum number of time values necessary todetermine whether the uncharacterized IA is an instance one of thecharacterized IA's.
 2. The system of claim 1, wherein the the at leastone computer program is configured to identify the IA in response toachievement of a predetermined correlation count, wherein each countcorresponds to a correlation between corresponding segments of the ERDtemplates of the characterized IA's and the updated ERD table of theuncharacterized IA.
 3. The system of claim 2, wherein the at least onecomputer program is configured to advance the correlation count inresponse to a minimum distance being recorded between correspondingsegments of the ERD templates and updated ERD table of the characterizedIA's and the uncharacterized IA respectively.
 4. The system of claim 1,further comprising the user device, wherein the user device includes arobotic arm or device, a full or partial limb prosthesis, an orthoticdevice, an electrical stimulation device, or an interface to a virtualdevice.
 5. The system of claim 1, wherein the at least one computerprogram is configured to initiate the device action instruction beforethe expiry of a pre-motor phase of a user action corresponding to theIA.
 6. The system of claim 1, wherein the ERD signal is anelectroencephalographic (EEG) or an electrocorticographic (ECoG) signal,a magnetic signal, or a chemical signal.
 7. The system of claim 1,wherein the at least one input includes a single electrode.
 8. A methodfor translating analog event-related desynchronization (ERD) signalsfrom a user into an identifiable intended activity (IA) to form one ormore ERD templates for use in associating an updated ERD table of anuncharacterized IA therewith to identify and initiate a correspondingintended device action by a user device, the method comprising: a.recording one or more ERD signals from one or more electrodesoperatively placed on the user, wherein each of the ERD signalscorresponds to a characterized intended activity (IA) of the user foreach time value of one or more successive time values; b. for each timevalue, forming a temporo-spectral decomposition of the ERD signal, toform a plurality of time resolved frequency ERD signal values; c.associating each ERD signal value with a function approximating asynthetic ERD signal to form at least one ERD table of coefficientscollectively representative of the characterized IA; d. repeating step cfor a number of instances of the characterized IA to form a number of atleast one ERD tables; and e. forming the ERD template for thecharacterized IA from the number of ERD tables for the characterized IA.9. The method of claim 8, wherein the recording of the one or more ERDsignals includes collecting one or more electrode signals from one ormore electrodes, with one or more of the ERD tables being associatedwith a corresponding electrode signal.
 10. The method of claim 8,wherein the time values correspond to overlapping or non-overlappingtime intervals.
 11. The method of claim 8, further comprising: a.recording the ERD signal corresponding to an uncharacterized IA for anumber of successive time values and updating an ERD table for theuncharacterized IA for each time value; and b. associating the updatedERD table for the uncharacterized IA with the ERD template of thecharacterized IA to determine whether the uncharacterized IA is aninstance of the characterized IA.
 12. The method of claim 11, furthercomprising issuing a device action instruction after a minimum number oftime values necessary to determine whether the uncharacterized IA is aninstance of the characterized IA.
 13. The method of claim 12, furthercomprising affirming that the uncharacterized IA is an instance one ofthe characterized IA's in response to achievement of a predeterminedcorrelation count, wherein each count corresponding to a correlationbetween corresponding segments of the updated ERD table of theuncharacterized IA and the ERD template of the characterized IA.
 14. Themethod of claim 13, further comprising advancing the correlation countin response to a minimum distance being recorded between correspondingsegments of the updated ERD table and the ERD template.
 15. The methodof claim 14, wherein the device action instruction occurs before anexpiry of a pre-motor phase of an action corresponding to the affirmedIA.
 16. The method of claim 15, wherein the time duration between thedevice action instruction and the expiry of the pre-motor phase, isminimized and/or optimized.
 17. The method of claim 16, wherein the userdevice is one of a robotic arm or device, full limb prosthesis, partiallimb prosthesis, or a user treatment device, a neuroprosthesis orfunctional electrical stimulation (FES) device that actuates a paralyzedlimb, an orthotic device, or a virtual activity device.